Display device, remote control device to control display device, method of controlling display device, method of controlling server and method of controlling remote control device

ABSTRACT

A display device, a remote control device for the display device, a method of controlling the display device, a method of controlling the remote control device, and a method of controlling a server includes collecting at least one piece of situation information or receiving at least one piece of situation information from a remote control device, determining a current state of a user based on the collected or received at least one piece of situation information, instructing the remote control device to output a notification signal corresponding to the current state of the user, and generating the control command according to the current state of the user and providing a sense-based image service to the user according to the control command.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of Korean PatentApplication No. 10-2012-0116955, filed on Oct. 19, 2012 in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference.

BACKGROUND

1. Field

The following description relates to a display device, a remote controldevice for the display device, a method of controlling the displaydevice, a method of controlling the remote control device, and a methodof controlling a server.

2. Description of the Related Art

A display device is an output device for representing electrical signalsas visual information and displaying the visual information to a user,and may include a television (TV), a computer monitor, and variousmobile devices such as a smartphone, a tablet computer, etc.

A remote control device controls a display device from a distance. Auser may view video content using a display device at a close distancewhere the user may easily manipulate the display device or at a distancewhere the user necessarily moves in order to manipulate the displaydevice. The remote control device may enable the user to manipulate thedisplay device from a distance.

Examples of the remote control device include a remote controller. Theremote controller emits infrared light having a predetermined frequencyto a display device 200. The display device receives the infrared light,converts the infrared light into an electrical signal according to thefrequency of the infrared light, sends the electrical signal to aprocessor, and performs various operations such as channel change orvolume control of the display device.

Recently, various terminals having a wireless communication function,such as a smartphone, may be used as a remote control device. In thiscase, the remote control device or the display device includes awireless communication module to transmit and receive an electricalsignal over a wireless communication network according to various mobilecommunication standards such as Bluetooth, Wi-Fi, 3GPP, 3GPP2, or WiMax.A remote control device such as a smartphone generates a predeterminedsignal according to user touch and transmits the predetermined signal toa display device, and the display device generates a control commandaccording to the predetermined signal and performs channel change orvolume control according to the control command.

SUMMARY

Therefore, it is an aspect of the present disclosure to control adisplay device based on a current state of a person or an environment,such as feelings or actions of a user who uses a display device todisplay an image, such as a TV, or surrounding environments.

It is an aspect of the present disclosure to provide various sense-basedservices to a user by controlling a display device based on a currentstate of a user.

It is an aspect of the present disclosure to enable a remote controldevice or a display device to acquire state information to determinebody states or feelings of a user from external environments and toappropriately check body states or feelings of the user.

It is an aspect of the present disclosure to provide various servicesusing a display device based on feelings or actions of a user, tostimulate the user' senses, and to enable the user to undergo user'sexperience (UX) to expand emotional experience of the user.

It is an aspect of the present disclosure to determine a user state andto further acquire information about surrounding environments based onthe user state or to provide a predetermined service or a service usingthe further acquired information to the user based on the user state.

It is an aspect of the present disclosure to enable a user to feelpositive emotions, pleasure and impressions by arousing interest andcuriosity of the user who uses a display device.

It is an aspect of the present disclosure to provide content or servicessuitable for a user according to habits, patterns, and preference of auser when the user views a display device or when the user is locatednear the display device.

It is an aspect of the present disclosure to provide a display deviceincluding a self-learning function by updating content or services forstimulating sense according to user's reaction or preference.

Additional aspects of the disclosure will be set forth in part in thedescription which follows and, in part, will be obvious from thedescription, or may be learned by practice of the invention.

In accordance with an aspect of the present disclosure, a display deviceincludes a situation determination unit to determine a current state ofa user based on at least one piece of situation information receivedfrom a remote control device or collected by an information collector,and a controller to instruct the remote control device to output anotification signal corresponding to the current state of the user andto generate control information to provide a sense-based image serviceto the user according to the current state of the user. The remotecontrol device outputs the notification signal through a notificationunit.

The display device may further include a determination informationdatabase to store determination information to determine the currentstate of the user based on the situation information. In this case, thesituation determination unit may search the determination informationdatabase for determination information corresponding to the situationinformation and determine the current state of the user based on thedetermination information corresponding to the situation information ifthe determination information corresponding to the situation informationis present.

The display device may further include a control information databaseincluding control information of the remote control device or thedisplay device corresponding to the current state of the user. In thiscase, the controller may detect the control information of the displaydevice corresponding to the current state of the user by referring tothe control information database and control the display deviceaccording to the detected control information.

The controller of the display device may receive a user command from theuser through the remote control device according to the notificationsignal, correct the generated control information according to the usercommand, and generate new control information.

In this case, the display device may further include a memory to storethe new control information generated by the controller, and an updateunit to analyze a user manipulation pattern based on the stored controlinformation and to update the control information database based on theanalyzed result.

The information collector may include at least one of an optical sensorto sense light, an acoustic sensor to sense sound, a motion sensor tosense motion, a temperature sensor to sense a temperature, and abiosignal sensor to sense temperature, pulse, respiratory rate, or heartrate of a person. The information collector may include a facerecognizer to detect a face shape from an image, to detect featurepoints of a face from the detected face shape, to analyze the positionsof the feature points of the face, to generate face information andfacial expression information, and to collect situation information.

The notification unit may include at least one of a light emittingmodule to emit light, a vibration generation module to generatevibrations and an acoustic output module to output sound.

The control information to control the display device may control acamera of the display device to capture an image, control the imagecaptured by the camera of the display device to be displayed on thedisplay device in a picture-in-picture (PIP) manner, control a thumbnailimage of at least one image captured by the camera of the display deviceto be displayed on a screen of the display device, or controlinformation about an image displayed on the screen of the display deviceto be displayed on the screen of the display device.

The display device may further include a beam projector module toproject an image onto an external projection surface.

The situation determination unit may aggregate situation information andimage information of an image played back by the display device anddetermine the current state of the user.

In accordance with an aspect of the present disclosure, a remote controldevice to control a display device includes an image collector tocollect at least one piece of situation information, a situationdetermination unit to determine a current state of a user based on theat least one piece of situation information collected by the informationcollector, a notification unit to output a notification signalcorresponding to the determined current state of the user, and acontroller to generate control information to enable the display deviceto provide a sense-based image service to a user according to thecurrent state of the user and to transmit the control information to thedisplay device, after the notification signal is output.

The remote control device may further include a determinationinformation database to store determination information to determine thecurrent state of the user based on the situation information indicatingfeatures of a special situation of an external environment. At thistime, the situation determination unit may search the determinationinformation database for determination information corresponding to thecollected situation information and determine the current state of theuser based on the determination information corresponding to thesituation information if the determination information corresponding tothe situation information is present.

The remote control device may further include a control informationdatabase including display control information corresponding to thecurrent state of the user; and a controller to detect the controlinformation of the display device corresponding to the current state ofthe user by referring to the control information database.

The remote control device may further include a command input unit toreceive a display manipulation command from the user after thenotification signal is output.

The controller of the remote control device may detect the controlinformation of the display device corresponding to the current state ofthe user by referring to the control information database and correctthe generated control information according to the user command receivedthrough the command input unit, and the remote control device mayfurther include a memory to store the control information generated bythe controller, and an update unit to analyze a user manipulationpattern based on the stored control information and to update thecontrol information database based on the analyzed result.

The information collector may include at least one of an optical sensorto sense light, an acoustic sensor to sense sound, a motion sensor tosense motion, a temperature sensor to sense a temperature, a biosignalsensor to sense temperature, pulse, respiratory rate, or heart rate of aperson, and a motion recognition sensor to motion of the remote controldevice. The information collector may include a face recognizer todetect a face shape from an image, to detect feature points of a facefrom the detected face shape, to analyze the positions of the featurepoints of the face, to generate face information and facial expressioninformation, and to collect situation information.

The notification unit may include at least one of a light emittingmodule to emit light, a vibration generation module to generatevibrations, and an acoustic output module to output sound.

The situation determination unit of the remote control device maydetermine the current state of the user based on the at least one pieceof situation information and image information of an image played backby the display device.

In accordance with an aspect of the present disclosure, a remote controldevice includes an image collector to collect at least one piece ofsituation information, and a notification unit to output a notificationsignal. In this case, the remote control device transmits the situationinformation collected by the information collector to a display deviceor a server, receives information about a current state of a userdetermined based on the collected situation information from the displaydevice or the server, and sends control information corresponding to thedetermined current state of the user or the collected situationinformation after the notification unit outputs the notification signalto control the display device.

The remote control device may further include a command input unit toreceive a display manipulation command from the user after thenotification signal is output and transmitting the display manipulationcommand to the display device.

The information collector of the remote control device includes at leastone of an optical sensor to sense light, an acoustic sensor to sensesound, a motion sensor to sense motion, a temperature sensor to sense atemperature, and a biosignal sensor to sense temperature, pulse,respiratory rate or heart rate of a person. The information collectormay include a face recognizer to detect a face shape from an image, todetect feature points of a face from the detected face shape, to analyzethe positions of the feature points of the face, to generate faceinformation and facial expression information, and to collect situationinformation.

The notification unit may include at least one of a light emittingmodule to emit light, a vibration generation module to generatevibrations, and an acoustic output module to output sound.

The situation determination unit may determine the current state of theuser based on the situation information and image information of animage played back by the display device.

In accordance with an aspect of the present disclosure, a method ofcontrolling a display device includes collecting at least one piece ofsituation information or receiving at least one piece of situationinformation from a remote control device, determining a current state ofa user based on the collected or received at least one piece ofsituation information, instructing the remote control device to output anotification signal corresponding to the current state of the user, andgenerating a control command according to the current state of the userand providing a sense-based image service to the user according to thecontrol command.

The situation information may be collected by sensing light andmeasuring the amount of light, measuring the volume of sound, sensingmotion, measuring a temperature, measuring a pulse or temperature of aperson, or measuring time information received from a satellite or acontroller.

The collecting the situation information may include detecting a faceshape from an image, detecting feature points of a face from thedetected face shape, and analyzing the positions of the feature pointsof the face, and collecting the situation information including faceinformation and facial expression information of the image.

The determining the current state of the user may include searching adetermination information database including determination informationto determine the current state of the user based on the situationinformation, and comparing the collected situation information with thedetermination information of the determination information database anddetermining the current state of the user from the determinationinformation if the determination information corresponding to thecollected situation information is present.

The sense-based image service may be provided by capturing an imageusing a camera of the display device, displaying the image captured bythe camera of the display device on the display device in apicture-in-picture (PIP) manner, displaying a thumbnail image of atleast one image captured by the camera of the display device on a screenof the display device, displaying information about an image displayedon the screen of the display device on the screen of the display device,or enabling a beam projector module of the display device to project animage onto a projection surface.

The generating the control command according to the current state of theuser may include searching a control information database includingcontrol information of the display device corresponding to the currentstate of the user, detecting the control information of the displaydevice corresponding to the current state of the user, and generatingthe control command according to the control information.

The determining the current state of the user may include determiningthe current state of the user based on the situation information andinformation about an image played back by the display device.

In accordance with an aspect of the present disclosure, a method ofcontrolling a display device includes collecting at least one piece ofsituation information or receiving at least one piece of situationinformation from a remote control device, determining a current state ofa user based on the collected or received at least one piece ofsituation information, instructing the remote control device to output anotification signal corresponding to the current state of the user, andreceiving a user command from the user according to the notificationsignal output from the remote control device and providing a sense-basedimage service to the user according to the collected or receivedsituation information and the control command.

In accordance with an aspect of the present disclosure, a method ofcontrolling a server includes, at a server, receiving at least one pieceof situation information from the display device or a remote controldevice, determining a current state of a user based on the receivedsituation information, instructing the remote control device to output anotification signal corresponding to the current state of the user, andtransmitting a control command to provide a sense-based image service tothe user according to the current state of the user and enabling thedisplay device to provide the sense-based image service to the useraccording to the control command.

In accordance with an aspect of the present disclosure, a method ofcontrolling a remote control device includes collecting at least onepiece of situation information, determining a current state of a userbased on the collected situation information, outputting a notificationsignal corresponding to the current state of the user, receiving a usercommand from the user according to the notification signal output fromthe remote control device, and generating a control command of a displaydevice based on the collected situation information and the usercommand, transmitting the control command to the display device andenabling the display device to provide a sense-based image service tothe user according to the control command.

In accordance with an aspect of the present disclosure, a method ofcontrolling a remote control device includes collecting at least onepiece of situation information, determining a current state of a userbased on the collected situation information, outputting a notificationsignal corresponding to the current state of the user, and generating acontrol command of a display device based on the collected situationinformation, transmitting the control command to the display device andenabling the display device to provide a sense-based image service tothe user according to the control command.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects of the disclosure will become apparent andmore readily appreciated from the following description of theembodiments, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 is a diagram showing a display device control system including adisplay device and a remote control device to control the display deviceaccording to an embodiment of the present disclosure;

FIG. 2 is a block diagram showing a display device and a remote controldevice according to an embodiment of the present disclosure;

FIG. 3 is a block diagram showing an information collector of a displaydevice according to an embodiment of the present disclosure;

FIG. 4A is a block diagram of a remote control device according to anembodiment of the present disclosure;

FIG. 4B is a front view of a remote control device according to anembodiment of the present disclosure;

FIG. 5 is a diagram showing the configuration of a system including adisplay device, a remote control device and a server according to anembodiment of the present disclosure;

FIG. 6 is a block diagram of a display device, a remote control device,and a server according to an embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating a method of controlling a displaydevice based on a situation according to an embodiment of the presentdisclosure;

FIG. 8 is a flowchart illustrating a method of controlling a displaydevice based on a situation according to an embodiment of the presentdisclosure;

FIG. 9 is a flowchart illustrating a method of controlling a displaydevice based on a situation according to an embodiment of the presentdisclosure;

FIG. 10 is a flowchart illustrating a method of controlling a displaydevice based on a situation according to an embodiment of the presentdisclosure;

FIG. 11 is a flowchart illustrating a method of controlling a displaydevice based on a situation according to an embodiment of the presentdisclosure;

FIG. 12 is a diagram illustrating a state in which a display device iscontrolled according to a method of controlling the display device basedon a situation according to an embodiment of the present disclosure;

FIG. 13 is a diagram illustrating a state in which a display device iscontrolled according to a method of controlling the display device basedon a situation according to an embodiment of the present disclosure;

FIG. 14 is a diagram illustrating a state in which a display device iscontrolled according to a method of controlling the display device basedon a situation according to an embodiment of the present disclosure; and

FIG. 15 is a flowchart illustrating a method of recognizing a user'sface and controlling a display device according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to the embodiments, examples ofwhich are illustrated in the accompanying drawings, wherein likereference numerals refer to the like elements throughout. Theembodiments are described below to explain the present disclosure byreferring to the figures.

Hereinafter, a display device and a remote control device to control thedisplay device according to an embodiment of the present disclosure willbe described with reference to FIGS. 1 to 6.

FIG. 1 is a diagram showing a display device control system including adisplay device and a remote control device to control the display deviceaccording to an embodiment of the present disclosure.

The display device control system of the embodiment of the presentdisclosure includes a remote control device 100 and a display device 200which may be controlled by the remote control device 100. The remotecontrol device 100 and the display device 200 may transmit and receive acommand or data to and from each other over a wireless communicationnetwork using various mobile communication standards such as Bluetooth,Wi-Fi, or 3GPP, 3GPP2, or WiMax.

The remote control device 100 may remotely control the display device200, e.g., a TV.

In an embodiment of the present disclosure, the remote control device100 is a remote controller which has predetermined buttons formed on onesurface or multiple surfaces thereof and emits infrared light having apredetermined frequency to the display device 200 to control the displaydevice 200 according to a button manipulated by a user.

According to an embodiment of the present disclosure, the remote controldevice 100 may be a terminal including a wireless communication module,which has a touchscreen or predetermined buttons formed on an outersurface thereof, generates a predetermined control signal according tomanipulation of a physical button or the touchscreen by a user, sendsthe generated control signal to the display device 200 over variouswireless communication networks, and controls the display device 200,such as a remote controller including a wireless communication module ora smartphone.

All kinds of remote control devices to control the display device 200 ata predetermined distance may be used as the remote control device 100 ofthe embodiment of the present disclosure.

In an embodiment of the present disclosure, the display device 200 mayoutput an image or an image and sound to provide video and/or audiocontent to a user.

The display device 200 may include an infrared reception unit to receiveinfrared light emitted from the remote control device 100 to becontrolled by the remote control device 100 or a transmission unit totransmit a predetermined control command from the remote control device100, such as a radio communication chip, according to embodiments of thepresent disclosure.

Various display devices which may be controlled by the remote controldevice 100 may be included in an embodiment of the present disclosure.For example, the display device 200 may be a TV or variousimplementations of an electronic image display device, e.g., arefrigerator display mounted in a housing of a refrigerator. Inaddition, examples of the display device include home appliances todisplay an image through a screen, e.g., a computer, a game console, asmartphone, etc. A beam projector module to project light such as laserbeams onto a projection surface and to display an image and a screen maybe included in the display device according to the embodiments of thepresent disclosure.

According to an embodiment of the present disclosure, external situationinformation is collected using the remote control device 100 and thedisplay device 200, the current state of a user or a surroundingenvironment is checked based on the collected information, and a controlcommand of the display device 200 is generated based on the checkedcurrent state to control the display device 200. According to anembodiment of the present disclosure, the control command of the remotecontrol device may be controlled.

Hereinafter, an embodiment of a system including the remote controldevice 100 and the display device 200 will be described with referenceto FIG. 2.

FIG. 2 is a block diagram showing a display device control systemincluding a remote control device according to an embodiment of thepresent disclosure.

Referring to FIG. 2, the remote control device 100 according to theembodiment of the present disclosure includes, for example, aninformation collector 110 to collect situation information from anexternal environment using various kinds of sensors and a notificationunit 140 to output light or sound and notify a user of the situationinformation. Hereinafter, the information collector 110 of the remotecontrol device 100 is referred to as a first information collector 110in order to be distinguished from an information collector 210 of thedisplay device 200.

The display device 200 may include a situation determination unit 220(hereinafter, the situation determination unit of the display device 200is referred to as a second situation determination unit in order to bedistinguished from a situation determination unit 120 of the remotecontrol device 100) to determine the current state of the user based onthe external situation information collected by the first informationcollector 110 of the remote control device 100 and a controller 230(hereinafter, the controller of the display device 200 is referred to asa second controller in order to be distinguished from a controller 130of the remote control device 100) to transmit the determination resultof the second situation determination unit 220 to the remote controldevice 100 or generate a control command according to the determinationresult and to control the display device 200.

According to an embodiment of the present disclosure, the display device200 may further include an information collector 210 (hereinafter,referred to as a second information collector in order to bedistinguished from the first information collector of the remote controldevice 100) to collect situation information from an externalenvironment. In this case, the second situation determination unit 220determines a situation using the situation information collected usingthe second information collector 210 of the display device 200 ordetermines the current state using the collected situation informationfrom the information collectors 110 and 210 of the remote control device100 and the display device 200.

In the embodiment of the present disclosure, as shown in FIG. 2, thedisplay device 200 may include a determination information database 221in which information for determination of the external situation of thesecond situation determination unit 220 is stored or a controlinformation database 231 in which information for generation of thecontrol command of the display device 200 by the second controller 230is stored. According to the embodiment, the display device 200 may beconnected to an external determination information database 221 orcontrol information database 231 over a wired/wireless communicationnetwork.

According to the embodiment, the display device 200 may further includea display unit 240 to display a variety of audio/video content to auser, a memory 250 to store the information collected by the secondinformation collector 210, the control command of the controller 230 orimage data captured by a camera 270 mounted on a front surface, a sidesurface, or a rear surface of the display device 200, and an update unit260 to analyze a manipulation pattern of the user based on the storedinformation and to learn the analyzed result.

As shown in FIG. 2, according to the embodiment, the display device 200may further include a camera module 270 mounted in the display device200 to capture an image or a projector module 280 to project beams to anexternal projection surface and to display an image on the projectionsurface.

Accordingly, in the display device control system including the displaydevice and the remote controller of the present disclosure, thesituation information collected by the information collector 110 or 210of the remote control device 100 or the display device 200 istransmitted to the display device 200 and the display device 200determines the situation based on the collected situation information,generates a control command corresponding thereto, and controls thedisplay device 200.

According to an embodiment, as shown in FIG. 4A, the situationdetermination unit 120 to determine the situation information and thecontroller 130 to generate the control command are included in theremote control device 100 to determine the situation and generate thecontrol command.

Hereinafter, as shown in FIG. 2, components will be described in theembodiment in which the situation determination unit 220 and thecontroller 230 are included in the display device 200.

First, the information collectors 110 and 210 to collect the situationinformation in the embodiment of the present disclosure will bedescribed with reference to FIGS. 3 and 4A.

In the embodiment of the present disclosure, the information collectors110 and 210, that is, the first information collector 110 and/or thesecond information collector 210, collect the situation information ofthe external environment, that is, situation information a1 of a userbody state or situation information a2 of a surrounding environment.

More specifically, various conditions of the external environment, forexample, light, sound, temperature, or motion are sensed and convertedinto digital data, information from an external device, such as a globalpositioning system (GPS) received via communication with a satellite, isreceived, or information about a played-back program is received over awired/wireless communication network, thereby acquiring a variety ofsituation information.

The situation information a1 of the user body state includes a varietyof information indicating the user body state, such as pulse,respiratory rate, or temperature, for example.

The situation information a2 of the surrounding environment includesinformation about various surrounding environments or a played-backprogram, such as the amount of light, ambient sound, the number ofpersons around the device, information about a program played back bythe display device 200, time information, etc.

The information collectors 110 and 210 of the embodiment of the presentdisclosure may be included in only one of the remote control device 100or the display device 200, or may be included in both the remote controldevice 100 and the display device 200.

The information collectors 110 and 210 may be designed to be randomlyarranged at an arbitrary position of the remote control device 100 orthe display device 200 or at an optimal position to sense thesurrounding environments, the situation information of which will beacquired, such as light, sound, vibrations, or various biosignals.

According to the embodiment of the present disclosure, the informationcollectors 110 and 220 may be designed such that the user recognizes apresence/absence of the sensor.

According to an embodiment of the present disclosure, as shown in FIGS.3 and 4A, the first information collector 110 of the remote controldevice or the second information collector 210 of the display device 200may include at least one of biosignal sensors 111 and 211, opticalsensors 112 and 212, acoustic sensors 113 and 213, motion sensors 114and 214, temperature sensors 115 and 215, and face recognizers 117 and216, and may further include an information collector such as a sensorwhich is not described or shown, as necessary.

The biosignal sensors 111 and 211 sense the user body state and collectthe situation information a1 of the user body state.

Here, the situation information a1 of the user body state refers toinformation about the body state of a person, e.g., a viewer who viewsthe display device. In other words, the situation information a1 of theuser body state refers to a variety of information indicating whetherthe user sleeps, is tired, or is excited.

More specifically, the situation information a1 of the user body statemay include, for example, biological information such as temperature,pulse, respiratory rate, neural oscillation, or heart rate.

The biosignal sensors 111 and 211 collect the situation information a1of the user body state. The biosignal sensors 111 and 211 are generallymounted in the remote control device 100 located near the user to easilyand accurately measure the biosignal of the user.

In this case, the biosignal sensor 111 may be mounted at an arbitraryposition of the remote control device 100 or at an optimal position tosense the biosignal.

According to an embodiment of the present disclosure, the biosignalsensors 111 and 211 may be a heat sensor mounted at a specific positionof the remote control device 100, for example, in a grip of the remotecontrol device 100, to measure the temperature of a person or a pulsesensor mounted in a portion of the user body, for example, an end of afinger of a person, to measure the pulse of the person.

The biosignal sensors 111 and 211 may be mounted in the display device100 according to the situation information a1 of the user body state tobe acquired, as shown in FIG. 2.

The optical sensors 112 and 212 sense light and collect the situationinformation a2 of the surrounding environment.

The information a2 of the surrounding environment refers to variousenvironment conditions around the remote control device 100 or thedisplay device 200, for example, presence of light, the amount of light,presence and volume of sound, and presence and motion of an externalobject.

The optical sensors 112 and 212 sense light in the surroundingenvironment to measure the amount of light or sense a presence of anexternal object, motion or shape of the object, convert the sensedresult into an electrical signal, and acquire the situation informationa2 of the surrounding environment. For the optical sensors 112 and 212,a photodiode or a phototransistor to convert light into an electricalsignal may be used or an infrared light emitting diode to convertinfrared light into an electrical signal may be used.

According to the embodiment of the present disclosure, the opticalsensors 112 and 212 may further include a light emitting module toirradiate visible light or infrared light toward a target such that theoptical sensors 112 and 212 receive light reflected from the target anddetect the target.

The acoustic sensors 113 and 213 sense and measure voice or sound aroundthe display device 200 and acquire the situation information a2 of thesurrounding environment.

The acoustic sensors 113 and 213 sense a sound wave having apredetermined frequency delivered via air using a transducer, such asaudible sound, and convert the sound into an electrical signal, therebyacquiring the situation information a2 of a variety of voice or sound ofthe surrounding environment, such as voice of persons.

Examples of the acoustic sensors 113 and 213 include a microphone.

The motion sensors 114 and 214 sense movement of the device. Forexample, the motion sensors 114 and 214 sense movement of the remotecontrol device 100 or the display device 200 in which the motion sensor114 or 214 is mounted and convert the sensed result into an electricalsignal.

For the motion sensors 114 and 214, for example, an accelerometer, atilt sensor, a rotation sensor, a gyroscope, etc. may be used.

The motion sensors 114 and 214 may sense whether the user shakes ormoves the remote control device, for example, the remote controller, andsense in which direction the remote controller is moved.

Because the display device 200 is fixed at a predetermined position, themotion sensors 114 and 214 are generally mounted in the remote controldevice 100 manipulated by the user.

The temperature sensors 115 and 215 measure heat of ambient air or anobject and measure temperature. For the temperature sensors 115 and 215,various sensors to measure the external temperature, such as a platinumresistance temperature detector, a thermistor or a thermocouple, may beused.

The remote control device 100 or the display device 200 may sensewhether the periphery of the display device 200, e.g., the room in whichthe display device 200 is located, is warm or cold using the temperaturesensors 115 and 215.

In an embodiment of the present disclosure, the information collectors110 and 210 of the remote control device 100 and the display device 200may include face recognizers 117 and 216.

The face recognizer 117 or 216 recognizes a face from the image receivedfrom the camera 270 mounted in the remote control device 100 or thedisplay device 200 and collects the situation information of the user.

In an embodiment of the present disclosure, the face recognizer 216 ofthe display device 200 first detects the face from the image receivedfrom the camera 270 and then detects feature points of the detectedface, e.g., a pupil or a mouth corner. After the feature points aredetected, the positions of the feature points and information aboutfacial expression or the shape of the face based on the predefinedpositions of the feature points are compared to analyze the featurepoints and the situation information including face information andfacial expression information of the face included in the image aregenerated.

In an embodiment of the present disclosure, the information collector110 or 210 may include a pressure sensor 116.

The pressure sensor 116 measures pressure applied to the sensor.Examples of the pressure sensor include a conductive rubber sensor tomeasure pressure according to deformation of conductive rubber based onpressure applied to conductive rubber.

The pressure sensor 116 may include a pressure sensor using conductiverubber, a pressure sensitive polymer or a piezoelectric polymeraccording to a pressure sensing material and include a centralizedpressure sensor or a distributed pressure sensor according to detectionmethod.

The pressure sensor 116 may be generally mounted in the remote controldevice 100 which is frequently used by the user or may also be mountedin the display device 200 as necessary.

In an embodiment of the present disclosure, the second informationcollector 210 may further include the camera module 270.

The camera module 270 captures an external target. More specifically,the camera module 270 senses external visible light and converts thevisible light into an electrical signal, thereby acquiring a fixed ormoving image.

The image received through the camera module 270 is stored in the memory250 of the display device 200 such that the face recognizer 217recognizes the face or the image is displayed on the display unit 240 ofthe display device 200.

Although not shown, in an embodiment of the present disclosure, theinformation collector 110 or 210 of the remote control device 100 or thedisplay device 200 may further include various sensors to collect avariety of situation information in addition to the above-describedsensors 111 to 270.

For example, the information collector 110 or 210 may include anultrasonic sensor (not shown) to sense ultrasonic waves.

The ultrasonic sensor irradiates inaudible ultrasonic waves to anexternal target and senses ultrasonic echo waves reflected from thetarget, thereby collecting information about presence of the target, theshape of the target or motion of the target.

The information collector 110 or 210 collects time information outputfrom the processor of the remote control device 100 or the displaydevice 200 or received from an external satellite and enables thecontroller 130 or 230 to generate the control command which variesaccording to when the situation information is collected.

The information collector 110 or 210 may collect information aboutcontent played back on the screen of the display device 200, that is,the display unit 240, such as program guide information (EPG). The EPGmay be collected by receiving the EPG over a wired/wirelesscommunication network or reading user log information from the memory ofthe remote control device 100 or the display device 200.

The user log information is meta information such as records indicatingwhether one piece of content is played back or data related thereto andincludes, for example, history information such as the number of timesthat the user views the content, and the number of times that the userstops viewing the content or a last viewing time.

The situation information a1 of the user body state, e.g., presence of auser or an action of a user, and the situation information a2 of thesurrounding environment, e.g., the amount of light, a temperature, atime, the number of persons around the display device 200, presence ofan object between a user and the display device 200, a played-backprogram and a variety of information to estimate a user's emotionalstate, may be collected using the information collector 110 or 210.

As described above, the situation information a1 or a2 of the user bodystate or the surrounding environment collected by the informationcollector 110 or 210 is sent to the second situation determination unit220 of the display device 200 as shown in FIG. 2 or the first situationdetermination unit 120 of the remote control device 100 as shown in FIG.3 a and is used by each device 100 or 200.

Hereinafter, the components of the embodiment of the present disclosurein which the situation determination unit and the controller areincluded in the display device 200 of the system will be described withreference to FIG. 3.

The information collector 110 or 210 shown in FIGS. 2 and 3 among thecomponents of the display device 200 is described above. Hereinafter, inan embodiment of the present disclosure, the second situationdetermination unit 220 of the display device 200 will be described withreference to FIG. 3.

In an embodiment of the present disclosure, the second situationdetermination unit 220 determines the current state of the user based onthe collected situation information a1 and a2 of the externalenvironment and determines the current situation. That is, the secondsituation determination unit 220 determines the current state of theuser and the feelings of the user using a plurality of pieces ofsituation information collected via the second information collector210, e.g., the situation information a1 of the user body state such asthe temperature and pulse of the user, the amount of light, soundvolume, motion, user facial expression, information about a played-backprogram, or a current time.

At this time, the second situation determination unit 220 receives thesituation information collected from the external environment, e.g., theamount of light, temperature, sound volume or the body state of the useras an input value and substitutes the input value into a predeterminedsituation determination algorithm to obtain a result value, anddetermines the current state of the user or the surrounding environmentaccording to the result value.

In other words, the second situation determination unit 220 maydetermine the environment and state of the user.

For example, the second situation determination unit 220 may determinethe amount of light sensed by the optical sensors 112 and 212, thevolume of sound sensed by the acoustic sensors 113 and 213, whether theuser is sleeping according to the biosignal 211 of the user, or in whichenvironment the user is sleeping, such as whether the user is sleepingwith the lights off, for example.

In order to enable the second situation determination unit 220 todetermine the situation, a plurality of situation determinationalgorithms may be used according to the collected situation informationand the determination information. Several pieces of situationinformation may be used as an input value of one algorithm or one pieceof situation information may be used as an input value of the pluralityof situation determination algorithms. In addition, individualsituations are determined using several algorithms and the determinedindividual situations are aggregated to determine the overall situationof the remote control device 100 and the display device 200.

As in the above-described example, a determination as to whether theuser is awake or sleeping is made from the biosignal of the user using apredetermined algorithm, a determination as to whether an illuminationis turned on or off is made using the information about the amount oflight, a current time or a time which has elapsed after the illuminationis turned on is determined based on time information output from theprocessor, and then a determination as to whether the user does not goto sleep at night is made.

In an embodiment of the present disclosure, as shown in FIG. 2, thedetermination information database 221 for determination of the secondsituation determination unit 220 may be used.

The determination information database 221 stores determinationinformation to determine the current state of the user based on thesituation information.

The determination information refers to determination information whenthe situation information is within a predetermined range or has apredetermined value. For example, the determination information refersto information about the determination method according to the collectedsituation information, such as determining that the illumination isturned off if the amount of sensed light is within a predetermined rangeor determining that the user feels good if the user is smiling.

In this case, the second situation determination unit 220 searches thedetermination information database 221 for determination informationcorresponding to the situation information in association with thecollected situation information a1 and a2. If the determinationinformation corresponding to the situation information is present, adetermination as to the current state of the user is made according tothe determination information corresponding to the situationinformation. If determination information corresponding to the situationinformation is not present, the second situation determination unit 220finishes the determination and waits until other situation informationis collected.

According to the determination of the second situation determinationunit 220, various user states and situation determination results of thesurrounding environment, such as detection and recognition of the user,the number of persons located in front of the display device, variousfeelings of the user such as pleasure, happiness, gloom, sorrow,boredom, anger or fear, whether the user is sleeping or not, theintensity of illumination, or stress of the user.

The result determined by the second situation determination unit 220 issent to the second controller 230 as shown in FIGS. 2 and 3 and thesecond controller 230 of the display device 200 generates and sends acontrol command corresponding to the determination result to a target tobe controlled.

Hereinafter, in an embodiment of the present disclosure, the secondcontroller 230 of the display device 200 will be described.

In an embodiment of the present disclosure, the second controller 230 ofthe display device 200 generates a control command to control the remotecontrol device 100 and/or the display device 200, e.g., the display unit240 or the image projector module such as the beam projector module 280of the display device 200 according to the result of determining thecurrent state based on the collected situation information. In addition,the second controller 230 may generate a control command to control theoperations of the various components configuring the display device 200.

By controlling the display device 200, the display device 200 mayprovide the user with a service to stimulate a sense of the user basedon the state or feeling of the user or the surrounding environment.

According to the embodiment of the present disclosure, the secondcontroller 230 generates control information to control the remotecontrol device 100 to output a predetermined notification signalaccording to the determination result and transmits the controlinformation to the remote control device 100 as shown in FIG. 2. Thecontrol information may include information about the operation of thenotification unit 140 of the remote control device 100, e.g., thepattern of light emitted from the light emitting module (151 of FIG. 3a) of the notification unit 140, information about the notificationsignal such as information about sound output from the acoustic outputmodule 153 of the notification unit 140, etc. The notification unit 140of the remote control device 100 operates according to the controlinformation.

According to an embodiment of the present disclosure, the secondcontroller 230 may transmit only the determination result to the remotecontrol device 100. In this case, the remote control device 100 maydetermine a notification signal to be output based on the receiveddetermination result and control the notification unit 140, therebyoutputting a notification signal such as predetermined light or sound.

In an embodiment of the present disclosure, the second controller 230may generate a control command according to the determination result ofthe second situation determination unit 220 by referring to the controlinformation database 231, as shown in FIG. 2.

The control information database 231 may store the control informationof the remote control device 100 and/or the display device 200corresponding to the current state of the user determined based on thesituation information.

The control information may control the camera of the display device tocapture an image if the collected situation information corresponds to apredetermined condition or control the display device to display theimage captured by the camera of the display device in apicture-in-picture (PIP) manner. In addition, the control informationmay control a thumbnail image of at least one image captured by thecamera of the display device to be regularly arranged on the screen ofthe display device or may control information related to the imagedisplayed on the screen of the display device to be displayed on thescreen of the display device.

The control information may include a variety of control informationaccording to various types of services to stimulate the user's sensesthrough the display device 200.

In an embodiment of the present disclosure, the second controller 230searches the control information database 231 according to the currentstate of the user determined by the second situation determination unit220, reads control information of the remote control device 100 and/orthe display device 200 corresponding to the current situation if thecontrol information is present in the control information database 231,and generates a control command corresponding thereto. Alternatively, ifthe control information of the remote control device 100 and/or thedisplay device 200 corresponding to the current situation is notpresent, the second controller may finish generation of the controlcommand and wait until other situation information is received from thesecond situation determination unit 220.

In an embodiment of the present disclosure, the second controller 230may generate new control information according to a user commandreceived through an input unit of the remote control device 100, e.g.,various buttons (see 140 of FIG. 4A or 130 a of FIG. 4B) arranged on theremote controller or correct already generated control information togenerate new control information.

In particular, in this case, the input unit of the remote control device100 may receive a user command from the user after the notification unit140 outputs the notification signal to the user.

That is, if the user manipulates the remote control device 100 accordingto the notification information output from the notification unit 140,information about manipulation is transmitted to the second controller230 of the display device 200 and the second controller 230 generatesnew control information corresponding thereto.

For example, upon determining that the user may not sleep for a longtime at night based upon the situation information collected by thesecond controller 230, the second controller 230 may operate theprojector module 280 of the display device 200 and project beams suchthat a constellation is displayed on the ceiling of the room of theuser. At this time, when the user manipulates the remote control device100 and inputs a command to stop the operation of the projector module280, the second controller 230 may generate a new control command tostop the operation of the projector module 280 according to the inputuser command.

In addition, although, in an embodiment of the present disclosure, thesecond controller 230 may necessarily generate a control commandaccording to the determination result of the second situationdetermination unit 220, the second controller 230 may randomly generatea control command corresponding to the determination result in anembodiment. In other words, the controller 230 may or may not generatethe control command to control the remote control device 100 or thedisplay device 200 even when the determination result of the situationdetermination unit 220 is the same.

Thus, the user may unexpectedly receive a service using the displaydevice 200. Therefore, it may be possible to further stimulate theuser's senses.

In an embodiment of the present disclosure, the display device 200includes the display unit 240 to display various images received from anexternal device or stored in an internal memory according to the controlcommand of the second controller 230.

In an embodiment of the present disclosure, the display device 200 mayfurther include the memory 250 to store the above-described controlcommand or a variety of information collected by an external devicethrough an Internet network or an image captured through the camera 270according to the above-described control command, e.g., a variety ofmeta information of an image played back through the display device 200.The data stored in the memory 250 may be retrieved by the secondcontroller 230 and used for learning of the display device 200 asnecessary and may be displayed through the display unit 240.

According to an embodiment of the present disclosure, the display device200 may further include the update unit 260 to analyze a manipulationpattern of the user based on the information stored in the memory 250and to update the control information database 231 based on the analyzedresult.

The control information database 231 is established by a serviceprovider who provides a service using the display device control systemincluding the remote control device and does not include the individualcharacteristics of the users.

Accordingly, user manipulation, e.g., user manipulation through theinput unit 150 of the remote control device 100, is statisticallyanalyzed and a control command to control the display device 200 isupdated according to user reaction or preference with respect to anindividual situation. The control information database 231 is updated bythe update unit 260.

In other words, the display device 200 learns how the display device 200is controlled to provide a service according to user preference throughthe update unit 260.

In an embodiment of the present disclosure, the display device 200 mayfurther include the camera module 270 to capture an external image. Thecamera module 270 may capture a still image and a moving image.

In an embodiment of the present disclosure, the display device 200 mayfurther include the projector module 280 to project an image onto anexternal projection surface.

The projector module 280 projects light such as visible light or laserlight onto an external projection surface, e.g., a screen, a wall or aceiling to form a predetermined image on the projection surface.

Hereinafter, an example in which the remote control device 100determines a situation and generates a control command will be describedwith reference to FIG. 4A.

FIG. 4A is a block diagram of a remote control device according to anembodiment of the present disclosure.

As shown in FIG. 4A, the remote control device according to anembodiment of the present disclosure may include the first informationcollector 110, the first situation determination unit 120, the firstcontroller 130 and the notification unit 140. The first informationcollector 110 is described above.

The first situation determination unit 120 of the remote control device100 determines the current state of the user based on the situationinformation collected by the first information collector 110 similarlyto the second situation determination unit 220 of the display device200.

According to an embodiment of the present disclosure, the firstsituation determination unit 120 may determine the current state of theuser using only the first information collector 110 of the remotecontrol device or receive the situation information collected by thesecond information collector 210 of the display device 200 to determinethe current state of the user.

The determination result acquired by the first situation determinationunit 120 of the remote control device is transmitted to the displaydevice 200 according to an embodiment of the present disclosure suchthat the second controller 230 of the display device 200 generates acontrol command using the determination result.

According to an embodiment of the present disclosure, the determinationresult acquired by the first situation determination unit 120 of theremote control device is sent to the first controller 130 of the remotecontrol device 100.

In an embodiment of the present disclosure, the remote control device100 may further include the first controller 130 to generate apredetermined control command based on the determination result of thefirst situation determination unit 120.

The first controller 130 of the remote control device 100 generates acontrol command to control operation of the notification unit of theremote control device 100 according to the determination result andsends the control command to the notification unit 140. The firstcontroller 130 of the remote control device 100 generates a controlcommand to control the display device 200, e.g., a control command todrive the camera module 270 or the projector module 280, and send thecontrol command to the display device 200.

As described above, if the display device 200 includes the secondcontroller 230 and the second controller 230 generates a control commandto control operation of the notification unit 140 of the remote controldevice 100 according to the determination result and sends the controlcommand to the remote control device 100, the first controller 130 ofthe remote control device 100 need not generate the control command tocontrol the notification unit 140.

As described above, if the second controller 230 of the display device200 generates a control command of the display device 200, the firstcontroller 130 of the remote control device 100 does not need togenerate the control command of the display device 200.

For example, according to an embodiment of the present disclosure, thefirst controller 130 of the remote control device 100 generates acontrol command to control only the notification unit 140 of the remotecontrol device 100 based on the result of determining the current stateof the user using the situation information and sends the determinationresult to the display device 200 such that the second controller 230 ofthe display device 200 generates a control command to control thedisplay device 200 based on the determination result.

As described with reference to FIG. 2, the controller 110 may refer tothe control information database stored in the memory of the remotecontrol device 100 or accessed by the remote control device 100 over awireless communication network when generating the control command,according to the embodiment.

In an embodiment of the present disclosure, the remote control device100 further includes the notification unit 140.

The notification unit serves to output a predetermined notificationsignal corresponding to the current state of the user determined by thefirst situation determination unit 120 of the remote control device 100or the second situation determination unit 220 of the display device200.

In other words, when the situation determination units 120 and 220determine the state of the user and then the display device 200 iscontrolled according to the determination result, the user is informedthat a new service is provided through the display device 200.

The notification unit may request the user to manipulate the input unit150 of the remote control device 100 and to input a user commandnecessary to control the display device 200.

The notification unit 140 includes at least one of a light emittingmodule 141 to emit light, a vibration generation module 142 to vibratethe remote control device 100, and an acoustic output module 143 tooutput sound, in an embodiment of the present disclosure.

The light emitting module 141 controls a light source to emitpredetermined light according to a light emitting control commandreceived from the controller 130 or 230.

The light emitting module 141 includes a light source such as a lightbulb using a filament mounted on an outer surface of the remote controldevice 100 or a light emitting diode (LED) using electroluminescenceeffect and a circuit to control the light source.

According to an embodiment of the present disclosure, the light emittingmodule 141 may emit light having a color or energy which is changedaccording to the result of determining the current state of the userusing the situation information or may emit light in different patterns.For example, the color of emitted light or the flickering period ofemitted light may be changed according to the determination result. Thepattern of light may be changed according to the determination result.For example, red light is first flickered and yellow light is thenflickered after a predetermined time or the pattern of light may bechanged with time.

Accordingly, the user is intuitively aware of the determination resultgenerated by the remote control device 100 or the display device 200 orhow the display device 200 is controlled, through the pattern or form oflight emitted from the light emitting module 141.

In an embodiment of the present disclosure, the color of a part or theentirety of the remote control device 100 may be changed according tooperation of the light emitting unit 141.

According to an embodiment of the present disclosure, a part or theentirety of a housing of the remote control device 100 may be formed ofa transparent or semi-transparent material to transmit light and thelight emitting module 141 is placed on a lower side of a part or theentirety of the housing. If the amount or color of the light emittedfrom the light emitting module 141 is changed, the color of the part orthe entirety of the remote control device 100 may be changed.

In the embodiment of the present disclosure, the light emitting module141 may display different colors according to the state of the remotecontrol device 100.

In a state in which the remote control device 100 is not driven or in adefault state in which a predetermined control command is not sent fromthe remote control device 100 or the display device 200, the lightemitting module 141 emits gray or black light or does not emit lightsuch that the remote control device 100 appears gray or black.

If a predetermined control command is sent from the remote controldevice 100 or the display device 200, the light emitting module 141emits light having another color, e.g., a delicate color having a lowchrominance or brightness such that the color of the remote controldevice 100 is changed. The number of times that the color is changed isnot limited to one and may be changed according to a predeterminedpattern. The color to be changed may be randomly selected.

When a predetermined time, e.g., 3 to 4 seconds, has elapsed after thecolor of the remote control device 100 is changed, the display device200 starts to be controlled.

After the control of the display device 200 is started, the lightemitting module 141 of the remote control device 100 may continuouslyemit light having the same color until the control of the display device200 is finished, according to an embodiment of the present disclosure.Accordingly, as long as the display device 200 is continuouslycontrolled, the color of the remote control device 100 is maintained.

According to an embodiment of the present disclosure, when the controlof the display device 200 is started or when a predetermined time haselapsed after the control of the display device 200 is started, thelight emitting module 141 of the remote control device 100 may emit grayor black light or stop light emission. Then, the remote control device100 is switched to the default state such that the color of the remotecontrol device 100 becomes gray or black.

Accordingly, the user is intuitively aware of the determination resultgenerated by the remote control device 100 or the display device 200 orhow the display device 200 is controlled, through change in color of theremote control device 100.

The vibration generation module 142 generates vibrations with respect tothe remote control device 100 according to the result determined usingthe situation information.

The vibration generation module 142 may vibrate the remote controldevice 100 using a vibration motor mounted in the remote control device100, for example.

In the embodiment of the present disclosure, by changing the strength ofvibrations or the vibration pattern of the remote control device 100based on the determination result, the user may check the determinationresult and how the display device 200 is controlled.

The acoustic output module 143 outputs predetermined sound, music, orrecorded voice to the user according to the result determined using thesituation information.

In an embodiment of the present disclosure, the acoustic output module143 receives an acoustic output control command from the firstcontroller 130, selects at least one of a variety of acoustic datastored in the memory of the remote control device 100, such as voicedata, music data, or sound data, according to the control command, andoutputs the data through an output unit such as a speaker.

Accordingly, the user becomes aware the determination result or how thedisplay device 200 is controlled, through voice, music, or sound.

In an embodiment of the present disclosure, the remote control device100 may include the input unit 150 to receive a user command accordingto user manipulation and generate and send a control commandcorresponding thereto.

The input unit 150 may include, for example, a plurality of buttons 140a placed on an outer surface of the remote control device 100 or atouchscreen on which virtual buttons are displayed, as shown in FIG. 4Band receive a command from the user through the buttons.

According to an embodiment of the present disclosure, after thenotification unit 140 outputs a predetermined notification signal, theinput unit 150 may receive a user command corresponding to anotification signal from the user. In other words, the input unit 150 isswitched to a standby state to receive a command corresponding to thenotification signal from the user after the output of the notificationsignal and generates an input command if the user manipulates theplurality of buttons 140 a or the touchscreen of the input unit 150. Thegenerated input command is sent to the controller 140 or the displaydevice 200 such that the controller 140 or the display device 200performs a next operation.

According to an embodiment of the present disclosure, the display device200 may be controlled to perform predetermined operation even when theinput unit 140 does not receive a user command.

An example of the remote control device 100 is shown in FIG. 4B.

FIG. 4B is a front view of a remote control device according to anembodiment of the present disclosure.

Referring to FIG. 4B, in an embodiment of the present disclosure, theremote control device 100 may be a remote controller 100 a to controlthe display device 200.

The remote controller 100 a includes a first information collector 110,e.g., an optical sensor 112 a to sense light, and a notification unit140, e.g., an LED module 141 a to emit predetermined light to a user, asdescribed above, in an embodiment of the present disclosure. The remotecontroller 100 a may include a transmission unit to transmit a controlcommand to the display device 200, e.g., an infrared irradiation unit131. In addition, a plurality of buttons 150 a to receive apredetermined command from the user may be placed on the outer surfaceof the remote controller.

As described above, in an embodiment of the present disclosure, theremote controller 100 a measures the intensity of light through theoptical sensor 112 a to obtain situation information and sends thesituation information to the situation determination unit 110 or 210 ofthe remote control device 100 or the display device 200. Then, theremote control device 100 or the display device generates thedetermination result based on the collected situation information andenables the LED module 141 a of the remote controller 100 a to outputlight having a predetermined color in a predetermined pattern to theuser according to the determination result.

Thereafter, if the user presses any one of the plurality of buttons 150a of the remote controller 100 a, a predetermined control command isgenerated by the input unit of the remote controller 100 a and istransmitted to the display device 200 through the infrared irradiationunit 131. Then, the display device 200 performs operations correspondingto the transmitted control command.

Hereinafter, a system including a display device, a remote controldevice, and a server in an embodiment of the present disclosure will bedescribed.

FIG. 5 is a diagram showing the configuration of a system including adisplay device, a remote control device and a server according to anembodiment of the present disclosure, and FIG. 6 is a block diagram of adisplay device, a remote control device and a server according to anembodiment of the present disclosure.

As shown in FIGS. 5 and 6, in an embodiment of the present disclosure,the server 300 may be further included.

The server 300 receives the collected situation information from theremote control device 100 or the display device 200, determines thecurrent state from the situation information, and generates a controlcommand corresponding thereto.

According to an embodiment of the present disclosure, as shown in FIG.6, the server 300 may include a situation determination unit 310(hereinafter referred to as a third situation determination unit), acontroller 320 (hereinafter referred to as a third controller), a memory330, and an update unit 340.

In this case, as shown in FIG. 6, when the remote control device 100,the display device 200 or the remote control device 100 and the displaydevice 200 collect situation information a1 and a2 through theinformation collectors 110 and 210, the collected information istransmitted to the server 300 over a wired/wireless communicationnetwork.

Then, the third situation determination unit 310 determines thesituation using the collected situation information and sends thedetermination result to the third controller 320. The third controller320 generates a control command based on the determination result andtransmits the control command to the remote control device 100 and/orthe display device 200.

According to an embodiment of the present disclosure, a control commandm to control the notification unit 140 may be transmitted to the remotecontrol device 100 and a control command n to control the display unit220 or the projector module 230 may be transmitted to the display device200. In other words, the control command transmitted from the server 300to the remote control device 100 and the control command transmittedfrom the server 300 to the display device 200 may be different.

The other components of the remote control device 100, the displaydevice 200, and the server 300 are substantially equal to theabove-described components. The components may be partially modifiedaccording to the characteristics of the device.

Hereinafter, a method of controlling a display device based on asituation according to an embodiment of the present disclosure will bedescribed with reference to FIGS. 7 to 11.

FIG. 7 is a flowchart illustrating a method of controlling a displaydevice based on a situation according to an embodiment of the presentdisclosure.

As shown in FIG. 7, according to an embodiment of the presentdisclosure, the method of controlling the display device based on thesituation includes Operation S400 of collecting situation information,Operation S410 of determining the current state of the user, OperationS420 of outputting a notification signal at the remote control device100 according to the determination result, and Operation S430 ofcontrolling the display device 200 according to the current state of theuser.

According to an embodiment of the present disclosure, in Operation S400of collecting the situation information, the remote control device 100or the display device 200 collects situation information of user bodystate or a surrounding environment through the information collector 110or 210.

For example, various sensors of the remote control device 100 or thedisplay device 200 recognize facial expression, a point of gaze, orvoice of the user located a predetermined distance from the displaydevice 200 to recognize the body state or tone of the user or detectuser motion or pressure of the remote control device 100 applied by theuser, thereby acquiring the situation information.

The information collector 110 or 210 of the remote control device 100 orthe display device 200 may collect situation information while the useruses or does not use the display device 200. The situation informationof the user and the surrounding environment may be collectedcontinuously, periodically, or arbitrarily.

The current state of the user may be determined based on the collectedsituation information (operation S410).

In this case, the current state of the user may be determined bysearching the previously stored determination information database 221for information corresponding to the collected situation information orsubstituting the situation information into the previously storedsituation determination algorithm.

Therefore, it may be possible to determine emotional and body state ofthe user.

In an embodiment of the present disclosure, a process of determining thefeelings of the user or the surrounding environment will be described.

In order to determine the feelings of the user, first, the facialexpression of the user is recognized using the camera and the facerecognizers 117 and 216 of the display device 200 or the remote controldevice 100, user voice is collected using the acoustic sensors 113 and213, e.g., a microphone, the feelings of the user are determined basedon the collected situation information, and the determination result isgenerated. The determination result refers to various feelings of theuser, e.g., pleasure, gloom, anger, etc.

In order to detect whether the user is sleeping or not, the situationinformation of the user state and the surrounding environment isobtained through the biosignal sensor 111 or 211, the temperature sensor115 or 215 and the camera 270 of the remote control device 100 or thedisplay device 200. In this case, in order to increase accuracy of thedetermination as to whether the user is sleeping or not, the timeinformation output from the controller 130 or 230 or information aboutthe intensity of the light sensed by the optical sensor 212 may befurther used.

If the situation is determined, the remote control device 100 outputsthe notification signal according to the determination result (operationS420).

The notification signal may be light emitted from the light emittingmodule 141 of the remote control device 100, vibration of the remotecontrol device 100 through the vibration module 141 or voice, sound, ormusic output from the acoustic output module 143.

The notification signal may be displayed to the user by changing thecolor of the remote control device 100 as described above. That is, inthe remote control device 100 having the housing, a part or the entiretyof which is formed of a transparent or semi-transparent material totransmit light, the light emitting module 141 emits predetermined lightto the part or the entirety of the housing to change the color of thepart or the entirety of the remote control device 100. The userrecognizes that the color of the part or the entirety of the remotecontrol device 100 is changed.

After the notification signal is output, the display device 200 iscontrolled according to the current state of the user (operation S430).

If the user is excited while viewing a sports game through the displaydevice 200, the situation information is collected and it is determinedthat the user state is excitement and the camera 270 of the displaydevice 200 captures the image of the rejoicing user and the capturedimage is stored in the memory 250. In this case, the image displayedthrough the display device 200 and the image of the user may be combinedand displayed in a PIP manner.

If the user is bored, the image captured by the camera of the displaydevice or the image edited in the PIP manner may be controlled to bedisplayed. In this case, if many captured images of the user arepresent, a thumbnail image may be extracted from the images of the user,be regularly arranged and displayed on the screen of the display device.

While the user views video content such as a movie or drama through thedisplay device 200, the user may control a variety of image informationrelated to the image displayed on the screen 240 of the display device200, e.g., main character information, ratings, keyword, backgroundinformation, etc. at a predetermined position of the display device. Inthis case, prior to display of the image information, a selection buttonmay be displayed on the screen in order to receive a selection commandfrom the user. Even if the user views content first or a plurality oftimes, the same operation may be performed according to log information.

In addition, if the user views the same video content through thedisplay device a plurality of times, e.g., three times or more, theimage displayed by the display device 200 may be controlled to beautomatically stored.

Hereinafter, various examples of the method of controlling the displaydevice based on situation will be described.

FIG. 8 is a flowchart illustrating a method of controlling a displaydevice based on situation according to an embodiment of the presentdisclosure.

As shown in FIG. 8, the method of controlling the display device basedon the situation may be performed as follows.

The remote control device 100 or the display device 200 operates in astandby mode for the method of controlling the display device based onthe situation (operation S500). At this time, the display device 200 maydisplay predetermined video content and the remote control device 100may control the display device 200 according to user manipulation tochange the channel of the display device 200 or control volume.

For example, a user action or a surrounding environment is changed, thatis, the pulse or respiratory rate of the user is changed or theintensity of light of the room is increased or decreased, orpredetermined video content start to be played back. Alternatively, theuser may manipulate the input button of the remote control device 100 orthe display device 200 such that the remote control device 100 or thedisplay device 200 collects the situation information (operation S510).

Then, the remote control device 100 or the display device 200 starts tocollect the situation information (operation S520). That is, if thepulse or respiratory rate of the user is changed, predetermined videocontent is played back or the user inputs a situation informationcollection command, the information collector 110 or 210 of the remotecontrol device 100 or the display device 200 starts to collect thesituation information.

According to the embodiment of the present disclosure, the remotecontrol device 100 or the display device 200 may continuously collectsituation information regardless of user action or manipulation orchange in surrounding environment. In this case, the standby mode forthe method of controlling the display device based on the situation isnot necessary.

If the situation information is collected, the first controller 130 ofthe remote control device 100 compares the situation informationcollected by the remote control device 100 or the display device 200with the determination information of the determination informationdatabase (operation S530) to determine whether determination informationcorresponding to the collected situation information is present.

If determination information corresponding to the collected situationinformation is not present (operation S531), the remote control device100 may be switched to the standby mode operation S500 again or maycollect the situation information of the user action or the surroundingenvironment (operation S520).

If determination information corresponding to the collected situationinformation is present, the current state of the user, e.g., pleasure,happiness, gloom, sorrow, boredom, anger or fear, is determined(operation S532).

Then, the first controller 130 of the remote control device 100generates a predetermined control command according to the determinationresult. At this time, one control command is used to control thenotification unit 140.

The remote control device outputs the notification signal according tothe control command to control the notification unit 140 (operationS540). For example, the light emitting module 141 may emit light or theacoustic output module 142 such as a speaker may output voice or sound.As described above, the color of the part or the entirety of the remotecontrol device 100 may be changed.

A determination as to whether the determination result is transmitted tothe display device 200 is made (operation S550). If the secondcontroller 230 of the display device 200 generates a control command tocontrol the display device 200 according to the determination result,the determination result needs to be transmitted to the display device200 in order to generate the control command (operation S551). Suchtransmission may be performed through a wireless communication networksuch as a Bluetooth or Wi-Fi network.

The remote control device 100 generates a control command correspondingto the determination result.

According to the embodiment of the present disclosure, the remotecontrol device 100 may receive a user command from the user through theinput unit 150 after the notification unit 140 outputs the notificationsignal.

The user becomes aware that the display device 200 may be controlled bya separate control command through the notification signal. In otherwords, the user becomes aware that a service provided using the displaydevice 200 is present.

Thereafter, a determination as to whether the user receives the service,that is, whether the display device 200 is desired to be drivenaccording to predetermined operation by the control information database231, how the display device 200 operates or which operation among apredetermined scheduled operations is selected and performed is made.

If the user manipulates the remote control device 100, that is, pressesa predetermined button 150 a, the first controller 130 of the remotecontrol device 100 generates a control command according to usermanipulation.

In an embodiment of the present disclosure, the notification signal maybe output without receiving a command from the user and a controlcommand may be automatically generated before the notification signal isoutput or when a short time has elapsed after the notification signal isoutput.

If the control command is not generated, for example, if all controlcommands are generated by the controller 230 or 320 of the displaydevice 200 or the server, operation of the remote control device 100 isfinished (operation S560).

If the control command is generated (operation S560), a control commandcorresponding to the current state of the user according to thedetermination result is generated. This control command is used tocontrol operation of the display device 200 as described above. Such acontrol command may be a control command to manipulate the camera 270 ofthe display device 200 to capture the image of the user or a controlcommand to manipulate the beam projector module 280 in order to projecta beam onto a projection surface to generate a predetermined image(operation S561).

The control command generated by the remote control device 100 istransmitted to the display device 200 (operation S570) and the displaydevice 200 operates according to the control command (operation S580).

FIG. 9 is a flowchart illustrating a method of controlling a displaydevice based on situation according to an embodiment of the presentdisclosure.

As shown in FIG. 9, the method of controlling the display device basedon situation according to an embodiment of the present disclosure may beperformed as follows.

When the remote control device 100 and the display device 200 are in astandby mode (operation S600), if the user action or surroundingenvironment is changed (operation S610), the remote control device 100and/or the display device 200 senses the user action or the surroundingenvironment and collects situation information (operation S620).

Thereafter, the remote control device 100 transmits the collectedsituation information to the display device 200 using a wirelesscommunication network using various mobile communication standards, suchas Bluetooth, Wi-Fi, 3GPP, 3GPP2, or WiMax (operation S630).

The display device 200 determines the current state of the user usingthe situation information collected by the remote control device 100.

According to an embodiment of the present disclosure, the display device200 may use the situation information collected by the remote controldevice 100 or the situation information collected by the secondinformation collector 210 of the display device 200. In other words, acombination of the situation information collected by the remote controldevice 100 and the situation information collected by the display device200 may be used or only the situation information collected by any oneof the devices 100 and 200 may be selectively used.

In an embodiment of the present disclosure, the display device 200compares the determination information of the determination informationdatabase with the collected situation information to determine thecurrent state, by referring to the determination information database221 (operation S640).

In this case, if determination information corresponding to thecollected situation information is present in the determinationinformation database 221, the current state of the user may bedetermined (operation S641 and operation S642). If determinationinformation corresponding to the collected situation information is notdetected, the remote control device 100 or the display device 200 may beswitched to the standby mode to generate the control command accordingto the situation information.

If the current state of the user is determined, the second controller230 of the display device 200 sends the control command to output thenotification signal to the remote control device 100 according to thedetermination result. At this time, a wireless communication networkusing various mobile communication standards, such as Bluetooth, Wi-Fi,3GPP, 3GPP2, or WiMax may be used to transmit the control command. Then,the remote control device 100 outputs the notification signal accordingto the control command to output the notification signal (operationS650).

The second controller 230 of the display device 200 generates a controlcommand to control the display device 200 in correspondence with thecurrent state of the user (operation S660).

According to an embodiment of the present disclosure, such controlcommands may be sequentially generated before or after the remotecontrol device 100 outputs the notification signal through thenotification unit 140 of the light emitting module 141 such as the LEDmodule without a command received from an external device.

According to an embodiment of the present disclosure, the remote controldevice 100 may output the notification signal through the notificationunit 140, the user manipulates the input unit 150 of the remote controldevice 100 to receive the user command, and then the control command maybe generated according to the user command.

For example, if the LED module of the remote control device 100 emitslight in a predetermined pattern, it may be possible to inquire as towhether the user wishes to drive the display device 200 according topredetermined operation by the control information database 231, thatis, whether the user wishes to receive a service through voicesimultaneously or sequentially output to the user or text displayed onthe screen of the display device 200. At this time, the user maymanipulate the button 150 a of the remote control device 100 todetermine whether the service is received from the display device 200 orhow a part of the display device 200 is driven while the service isprovided.

The display device 200 is controlled automatically by the display device200 or according to a control command generated by manipulating theremote control device 100 by the user. For example, the display device200 may capture and display an image of a user or display a played-backvideo on a screen along with a previously captured image in a PIPmanner. In addition, the beam project module of the display device 200may be driven (operation S670).

FIG. 10 is a flowchart illustrating a method of controlling a displaydevice based on situation according to an embodiment of the presentdisclosure.

As shown in FIG. 10, an embodiment of the method of controlling thedisplay device based on situation will be described.

First, as described above, the remote control device 100 or the displaydevice 200 senses the action or situation of the user and collectssituation information (operation S700).

The situation information collected by the remote control device 100 orthe display device 200 is transmitted to the server 300 shown in FIG. 5over a wired/wireless communication network (operation S710).

Then, the server 300 determines the current situation using thesituation information collected by the remote control device 100 or thedisplay device 200. In this case, according to an embodiment of thepresent disclosure, the determination information database 221established in the server 300 or connected to the server 300 over thewired/wireless communication network is searched, the situationinformation and the determination information are compared and thedetermination information corresponding to the situation information isread (operation S720 and operation S721).

The current state of the user is determined based on the readdetermination information to generate a determination result (operationS722).

The server 300 transmits the determination result to the remote controldevice 100 (operation S730) and the remote control device 100 receivesthe determination result and outputs the notification signalcorresponding to the determination result (operation S731).

According to the embodiment of the present disclosure, the server 300may simultaneously or sequentially transmit the determination result tothe remote control device 100 and the display device 200 (operation S740and operation S741).

If the determination result is transmitted to the display device 200(“Yes” of operation S740), the display device 200 generates a controlcommand based on the determination result. In this case, the server 300does not generate the control command (“No” of operation S750). At thistime, the second controller 230 of the display device 200 generates thecontrol command corresponding to the current state of the user tocontrol the display device 200 as in operation S660 of FIG. 9.

If the determination result is not transmitted to the display device 200(“No” of operation S740 and “Yes” of operation S750), the server 300generates the control command according to the determination result(operation S751). According to the embodiments of the presentdisclosure, the control information database 231 established in theserver 300 or connected to the server over the wired/wirelesscommunication network may be referred to when generating the controlcommand.

Thereafter, the generated control command is transmitted to the displaydevice 200 (operation S752) and the display device 200 is controlled toprovide a service to the user according to the control command generatedby the server (operation S760).

FIG. 11 is a flowchart illustrating a method of controlling a displaydevice based on situation according to an embodiment of the presentdisclosure.

As shown in FIG. 11, an embodiment of the method of controlling thedisplay device based on the situation will be described.

As shown in FIG. 11, in an embodiment of the present disclosure, onlythe second information collector 210 of the display device 200 maycollect situation information (operation S800). For example, the facerecognizer 216 of the second information collector 210 of the displaydevice 200 acquires face information or facial expression informationfrom the face of the user located in front of the display device 200.

Then, the display device 200 compares the determination information ofthe determination information database 221 with the collected situationinformation, e.g., the face information or facial expression information(operation S810).

If the determination information corresponding to the collectedsituation information is present in the determination informationdatabase 221 (operation S820), the current state of the user isdetermined based on the determination information (operation S830).

Then, the display device generates a control command of the remotecontrol device 100 corresponding to the current state and a controlcommand of the display device 200 (operation S840).

The control command of the remote control device 100 is transmitted tothe remote control device 100 in order to control the notification unit140 of the remote control device 100 and the remote control deviceoutputs the notification signal corresponding to the current state ofthe user according to the control command (operation S850).

The display device 200 is controlled based on the control command of thedisplay device 200 (operation S860). For example, if it is determinedthe user is smiling as the result of determining the facial expressioninformation of the user, an image received during a predetermined timeor an image of at one moment in the facial image of the user is storedand processed to generate an image file of the user's face, therebycapturing the image of the user's face.

Hereinafter, various embodiments of a service provided to a user bycontrolling the display device according to the method of controllingthe display device based on situation will be described with referenceto FIGS. 12 to 15.

FIG. 12 is a diagram illustrating a state in which a display device iscontrolled according to a method of controlling the display device basedon a situation according to an embodiment of the present disclosure.

The user is viewing at a video such as a sports game or drama through adisplay screen p1 (FIG. 12( a)). In this case, the remote control device100 and the display device 200 may collect situation information throughthe information collectors 110 and 210.

If the user is excited as shown in FIG. 12( b), biological informationsuch as the pulse or respiratory rate of the user is collected throughthe biosignal sensors 111 and 211, a scream of the user is sensedthrough the acoustic sensor 113 to collect acoustic information, anduser actions are sensed through the optical sensor 112 to collect useraction information.

If the situation determination unit 120, 220 or 310 determines that theuser is excited based on the collected information, first, the remotecontrol device 100 outputs a notification signal. The camera module 270of the display device is driven to film the moving image of the userafter the user manipulates the remote control device 100 or withoutmanipulating the remote control device 100. The filmed moving image isstored in the memory 250.

Thereafter, a selection button is displayed at the lower side of theplayed-back image p1 under a predetermined situation (FIG. 12( c)).

The predetermined situation may follow previously stored settings suchas when a predetermined time has elapsed after the moving image isfilmed or when video content of the same genre is viewed.

The predetermined situation may be a situation that the user wishes toview the recorded image again, e.g., a situation that it is determinedthat the user is gloomy, based on the result determined based on thesituation collected by the first situation determination unit 120. Inorder to determine the situation, the above-described method ofcontrolling the display device based on the situation may be performed.In this case, the control command of the display device 200 is todisplay the selection button at the lower side of the played image p1.

Thereafter, the moving image p2 stored by manipulating the remotecontrol device 100 by the user is displayed along with the currentlyplayed-back video content p1 in a PIP manner. In this case, informationabout the played-back video, e.g., program guide information, may besimultaneously displayed as a subtitle (FIG. 12( d)).

Accordingly, the present disclosure may control the display device 200to provide a service to stimulate user's senses. Thus, the user may viewthe image of the user who views the sports game or drama after apredetermined time or according to their mood or feels positive emotionswhile remembering when viewing the sports game or drama.

FIG. 13 is a diagram illustrating a state in which a display device iscontrolled according to a method of controlling a display device basedon a situation according to an embodiment of the present disclosure.

As shown in FIG. 13, in the method of controlling the display devicebased on the situation according to an embodiment of the presentdisclosure, if the situation determination unit 120, 220, or 310determines that the video content p1 viewed by the user has beenrepeatedly viewed while viewing the video content p1 via the TV, theprogram guide information of the played-back video content and theviewing time may be stored.

The title of the content stored according to user selection may bedisplayed on the screen of the display device 200 (p1). In this case,the size of the title of the content may be changed according to thenumber of times that the user views the content or the viewingfrequency. For example, if the number of times that the user views thecontent or the viewing frequency is large, the size of the title may belarge and, if the number of times that the user views the content or theviewing frequency is small, the size of the title may be small (FIG. 13(b)).

At this time, the user manipulates the remote control device 100 toselect any one title and the display device 200 plays the videocorresponding to the selected title back on the display screen, therebyrapidly providing the user with the video having a high frequency ofviewing.

FIG. 14 is a diagram illustrating a state in which a display device iscontrolled according to a method of controlling a display device basedon situation according to an embodiment of the present disclosure.

As shown in FIG. 14, the user may not sleep at night.

At this time, according to an embodiment of the present disclosure, theremote control device 100 or the display device 200 collects situationinformation a1 of a user body state and situation information a2 of asurrounding environment. At this time, the situation information a1 ofthe user body state indicates that the user is awake but is stationaryand the situation information a2 of the surrounding environmentindicates that the intensity of light is equal to or less than apredetermined level, the time of day is night, e.g., 12 am, and a periodof time when the intensity of light is equal to or less than thepredetermined level exceeds 30 minutes.

The remote control device 100 or the display device 200 determines thatthe user does not go to sleep for a long time at night based on suchsituation information and generates a control command to control theremote control device 100 and the display device 200. In particular, thecontrol command to control the display device 200 includes a controlcommand to control driving of the beam projector module 280.

The remote control device 100 outputs a notification signal for apredetermined time according to the generated control command. At thistime, a part or the entirety of the remote control device 100 emitslight having a soft color using the light emitting module. In this case,the remote control device 100 may not receive an additional command fromthe user. In other words, the display device 200 may be automaticallycontrolled without manipulating the remote control device 100 by theuser, as described with reference to FIG. 11.

Thereafter, the beam projector module 280 of the display device 200projects a beam onto a projection surface, e.g., a ceiling, and displaysconstellations, a night sky, video content such as a movie or drama, apast image of the user, etc. on the projection surface, e.g., theceiling.

Thereafter, the user may move the remote control device 100 in apredetermined pattern or manipulate the input unit 150 of the remotecontrol device 100 to move the image projected on the projection surfacesuch as the ceiling. In this case, the motion sensor 114 of the remotecontrol device may be used to determine how the remote control device100 is moved. In other words, when the user moves or rotates verticallyor horizontally in a state in which the remote control device 100 isdirected to the projected image, the motion sensor 114 senses andconverts such movement or rotation into an electrical signal and thusthe first controller 130 of the remote control device 100 or the secondcontroller 230 of the display device 200 controls the beam projectormodule 280 such that the image projected onto the projection surface ischanged according to movement of the remote control device 130.

Thereafter, the situation collector 110 and 210 of the remote controldevice 100 or the display device 200 collects the situation informationof the user continuously or at predetermined frequency. If it isdetermined that the user is sleeping as the result of determining thecollected situation information, a control command of the beam projectormodule 280 is generated to stop operation of the beam projector module280.

Accordingly, if the user does not easily go to sleep at night,constellation or a past image of the user may be displayed on theprojection surface through the beam projector module 280 to enable theuser to go to sleep soundly. In addition, it may be possible to preventthe display device 200 and, more particularly, the projector module 280from meaninglessly operating.

In an embodiment of the present disclosure, the facial expression of theuser located near the image display may be recognized such that thedisplay device 200 captures the image of the user, which will bedescribed in greater detail with reference to FIG. 15.

FIG. 15 is a diagram illustrating a process of collecting situationinformation such as face information and facial expression informationin a method of recognizing a user's face and controlling a displaydevice according to an embodiment of the present disclosure.

Referring to FIG. 15, the process of acquiring the face information andthe facial expression information will now be described.

According to an embodiment of the present disclosure, first, the userand, more particularly, the face of the user approaches the camera 270.At this time, a plurality of users may be present (operation S900).Then, the optical sensor 212 of the display device 200 recognizesapproaching of a person and drives the camera 270 and the facerecognizer 216 (operation S910).

According to an embodiment of the present disclosure, the camera 270 andthe face recognizer 216 may start to operate according to usermanipulation without sensing approach of the user.

The camera 270 captures the image of the user's face and obtainstemporary image data including the face of at least one user, as the atleast user's face approaches (operation S920).

Thereafter, a face is detected from the captured image, that is, thegenerated image data (operation S930). According to the embodiment ofthe present disclosure, light and shade or a skin color is identifiedand pixels thereof are extracted to detect a portion corresponding to aface or extracted pixels are combined and compared with data of the faceshapes of a plurality of persons to detect a face from the capturedimage.

A feature point of the face, such as a pupil or a mouth corner, isdetected from the detected face (operation S940). In this case, thefeature point may be detected from the face based on the shape of thefeature point.

The position of the feature point of the face is compared with data ofthe face of the previously stored user or person to determine or analyzeto whose face the detected face corresponds or the facial expression(operation S950) and face information and facial expression informationare generated according to the determination result (operation S960).

If the second situation determination unit 210 determines that the useris smiling based on the generated face information and facial expressioninformation, the second controller 220 controls the camera 270 tocapture the image of the user according to the determination result,converts data obtained by capturing the image of the user into an imagefile, and stores the data in a separate memory.

Although not shown, according to the embodiment of the presentdisclosure, the remote control device 100 outputs a predeterminednotification signal according to the result of determining the situationand controls the display device such that information about a program isdisplayed at the lower side of the screen of the display device 200. Inthis case, the information displayed on the lower side of the screen maybe information collected through a web or program guide informationtransmitted along with the image.

The user may experience various services provided by controlling thedisplay device 200 based on the situation. The display device 200 mayprovide one or a plurality of services. The service may be changedaccording to a period of time when the user uses the display device orthe frequency of use thereof.

For example, the display device 200 is controlled such that a useraction is initially recorded to generate a moving image or the userrecords the frequency or time of using video content. At this time, whenthe user does not go to sleep at night, the display device 200 may becontrolled to display constellations on a ceiling as described above.

Thereafter, after a predetermined time has elapsed, the recorded useraction is played back or a variety of information about a program whichis viewed again, e.g., program guide information or ratings informationmay be provided.

According to an embodiment of the present disclosure, the controlcontent of the display device 200 or the result of determining thesituation may be recorded and may be arbitrarily searched by the userafter a predetermined period, e.g., 99 days, has elapsed. At this time,the recorded content may be displayed on the display screen in the formof a user interface such as a diary. Then, the user may confirm theiremotional state and how the display device 200 is controlled.

As described above, in the present disclosure, in controlling thedisplay device to provide a service, the service may be arbitrarily orrandomly provided to the user according to the state of the user or asnecessary. Then, the user receives an unexpected service to stimulatethe user's senses. Accordingly, the user obtains positive emotionaleffect through the display device.

According to the present disclosure, by providing a display device, aremote control device to control the display device and a method ofcontrolling the display device based on situation, it may be possible tocontrol the display device based on a current state of a person or anenvironment, such as feelings or actions of a user who uses a displaydevice or surrounding environments.

It may be possible to provide a user with various sense-based servicessuitable for feelings of the user and the surrounding environments basedon a current state of a user.

It may be possible to acquire a variety of state information todetermine body states or feelings of a user from feelings or actions ofthe user or external environments to accurately determine the state orfeelings of the user and to appropriately control the display deviceaccording the state of the user.

It may be possible to appropriately control the display device accordingto the state of the user to efficiently stimulate the user's senses, andto enable the user to be subjected to user's experience based on thestates, actions or feelings of the user. Therefore, the user feelspositive emotions through the display device.

It may be possible to provide a variety of content or services which mayarouse various emotions based on the feelings or actions of the user.

It may be possible to enable the display device to learn and improvecontent or services to stimulate user's senses according to user'sreaction or preference. Therefore, the user may receive optimal contentor services.

The above-described embodiments may be recorded in computer-readablemedia including program instructions to implement various operationsembodied by a computer. The media may also include, alone or incombination with the program instructions, data files, data structures,and the like. The program instructions recorded on the media may bethose specially designed and constructed for the purposes ofembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts. Examples ofcomputer-readable media include magnetic media such as hard disks,floppy disks, and magnetic tape; optical media such as CD ROM disks andDVDs; magneto-optical media such as optical disks; and hardware devicesthat are specially configured to store and perform program instructions,such as read-only memory (ROM), random access memory (RAM), flashmemory, and the like. The computer-readable media may also be adistributed network, so that the program instructions are stored andexecuted in a distributed fashion. The program instructions may beexecuted by one or more processors. The computer-readable media may alsobe embodied in at least one application specific integrated circuit(ASIC) or Field Programmable Gate Array (FPGA), which executes(processes like a processor) program instructions. Examples of programinstructions include both machine code, such as produced by a compiler,and files containing higher level code that may be executed by thecomputer using an interpreter. The above-described devices may beconfigured to act as one or more software modules in order to performthe operations of the above-described embodiments, or vice versa.

Although a few embodiments of the present disclosure have been shown anddescribed, it would be appreciated by those skilled in the art thatchanges may be made in these embodiments without departing from theprinciples and spirit of the invention, the scope of which is defined inthe claims and their equivalents.

What is claimed is:
 1. A display device comprising: a wirelesscommunicator configured to transmit and receive a command or data with aremote control device; a situation determination unit configured todetermine a current state of a user based on at least one piece ofsituation information received from the remote control device using thewireless communicator or collected by an information collector; and acontroller configured to generate control information to provide asense-based image service to the user according to the current state ofthe user by transmitting and receiving a command or data with the remotecontrol device using the wireless communicator, wherein the remotecontrol device outputs a notification signal, and is able to transmit auser command after outputting the notification signal, and wherein thecontroller generates the control information after the controllerreceives the user command from the remote control device.
 2. The displaydevice according to claim 1, further comprising a determinationinformation database to store determination information to determine thecurrent state of the user based on the situation information, whereinthe situation determination unit searches the determination informationdatabase for determination information corresponding to the situationinformation and determines the current state of the user based on thedetermination information corresponding to the situation information ifthe determination information corresponding to the situation informationis present.
 3. The display device according to claim 1, furthercomprising a control information database comprising control informationof the remote control device or the display device corresponding to thecurrent state of the user.
 4. The display device according to claim 3,wherein the controller detects the control information of the displaydevice corresponding to the current state of the user by referring tothe control information database and controls the display deviceaccording to the detected control information.
 5. The display deviceaccording to claim 1, wherein the controller corrects the generatedcontrol information according to the user command, and generates newcontrol information.
 6. The display device according to claim 5, furthercomprising: a memory to store the new control information generated bythe controller; and an update unit to analyze a user manipulationpattern based on the stored control information and to update thecontrol information database based on the analyzed result.
 7. Thedisplay device according to claim 1, wherein the information collectorcomprises at least one of an optical sensor to sense light, an acousticsensor to sense sound, a motion sensor to sense motion, a temperaturesensor to sense a temperature, and a biosignal sensor to sense at leastone of temperature, pulse, respiratory rate, and heart rate of a person.8. The display device according to claim 1, wherein the informationcollector comprises a face recognizer to detect a face shape from animage, to detect feature points of a face from the detected face shape,to analyze the positions of the feature points of the face, to generateface information and facial expression information, and to collectsituation information.
 9. The display device according to claim 1,wherein the notification unit comprises at least one of a light emittingmodule to emit light, a vibration generation module to generatevibrations, and an acoustic output module to output sound.
 10. Thedisplay device according to claim 1, wherein the control information tocontrol the display device performs at least one of: controlling acamera of the display device to capture an image, controlling the imagecaptured by the camera of the display device to be displayed on thedisplay device in a picture-in-picture (PIP) manner, controlling athumbnail image of at least one image captured by the camera of thedisplay device to be displayed on a screen of the display device, andcontrolling information about an image displayed on the screen of thedisplay device to be displayed on the screen of the display device. 11.The display device according to claim 1, further comprising a beamprojector module to project an image onto an external projectionsurface.
 12. The display device according to claim 1, wherein thesituation determination unit aggregates situation information and imageinformation of an image played back by the display device and determinesthe current state of the user.
 13. A remote control device comprising: awireless communicator configured to receive a command or data regardinga current state of a user determined based on situation informationcollected from a display device; a notification unit configured tooutput a notification signal corresponding to the received current stateof the user based on the command or data received by the wirelesscommunicator; and a controller configured to be able to transmit a usercommand to the display device using the wireless communicator after thenotification unit outputs the notification signal, wherein, after thedisplay device receives the user command, the display device generatescontrol information to provide a sense-based image service to the useraccording to the current state of the user.